I also use the FMC126-2-1-1-1 on a VC707 board. I did not observe any trouble with the original firmware. The console prog is called with "1 VC707 0 0 0" parameters. I use the ETHERNET link. My computer contains 2 ethernet boards, one is dedicated to the web access, the other to the vc707. I only found minor bugs in the console prog Fmc12xApp (diagnotics parsing) : in attached file, search for "!BUG!" text to locate them. Also in attached file you can check the memo detailing the results obtained. Do you obtain same console outputs ? I also modified the constellation and the UCF file to install the FMC126 on HPC2 and it also works (except adc3 due to HPC pins not routed to the FPGA).
Maybe your ADC is broken ?
About 4FM GUI, I cannot really answer. In my case, I have modified slightly the console prog so that all the displays are sent to a windows Form. data between console thread and WinForm use Name Shared Memory (NSM). It is more comfortable. The console has become a slave which executes commands sent by the Win Form GUI.But the code in the console remains the same.
About the CPLD programming, it is explained pages 18-19-20 of SD039 FMC126 STAR manual. See attached file for a summary. All the data are coming from 4DSP manuals and Arnaud/Erik's posts.