Jack: On Sun, 14 Apr 2024 at 22:18, jack <jack4pg@xxxxxxx> wrote: > The CSV files are being produced by another system, a WIndows app on a Windows machine. I then copy them to a USB key and copy them onto the ubuntu machine. The data is then imported via the COPY command. > > COPY master (field01,field02..fieldX) FROM '/data/file.text' DELIMITER E'\t' > The fields are tab delimited. Then call them tab-delimited, not CSV. It may be important. You have a tendency of sending your interpretation of information which hampers people trying to help you. As an example, only about three messages above did we see the "signal 11" in the server logs which indicates a probable bug. Signal 11 is normally due to stray pointers or corrupted memory ( although faulty ram can lead to it too ), which hints at a bug, not a resource problem. > But importing the data works. I can get all the data into a single table without any problems. The issue is only when I start to update the single table. And that is why I started using smaller temporary tables for each CSV file, to do the updates in the smaller tables before I move them all to a single large table. > After all the data is loaded and updated, I run php programs on the large table to generate reports. All of which works well EXCEPT for performing the updates on the data. And I do not want to use perl or any outside tool. I want it all one in SQL because I am required to document all my steps so that someone else can take over, so everything needs to be as simple as possible. I suggested perl ( or any similar thing, php should be equally fine ) as it normally makes your importing / debugging much easier / lighter. On the documentation side, you can have the data and some transformations in a big sql file, but having some steps in a php file will have them documented too. But if you want to do text processing in SQL, go ahead, the only problems are going to be making it harder to debug and probably harder to document. Now people know the signal 11 stuff and the problem can probably be tracked. Francisco Olarte.
