I have imported a SQL table that had 13 million records and 38 columns instantly through a JDBC connector. Conversely, I am trying to load a SQL table that has 90 million records and 219 columns through the same connector but it seems like Paxata is spinning its wheels trying to load the table. Is there a limitation on the table size that Paxata can import? Is there a setting that can be applied in the application or on the server side to bring in this large table?
Hi CONCJA, there are some theoretical limits to the size of an import, but there are likely other factors at play here that are causing the wheel spinning. Assuming a similar density (average amount of data per cell), the second table you're importing is ~40x larger than the first and should take proportionately longer to import. Other factors like bandwidth availability, Paxata core server utilization, and data source utilization can all have impacts as well. Did the import ever finish or did it fail?
Diagnosing these issues can be difficult without looking closely at the logs of Paxata and sometimes the data source as well. If you'd like help looking into this import, please reach out to your Paxata Customer Success contact and we can take a look with you.
I did wait a little longer for the 90 million records to load and Paxata was able to load the data. I believe it took over a half hour to load the data. I am curious @JoshCKF based on the factors you listed whether my organization's setup is at its optimal state to import such large datasets. I may reach out to Paxata Customer Service to review my organization's setup.