cancel
Showing results for 
Search instead for 
Did you mean: 

Is there a limitation on the table size that Paxata can import?

Highlighted
Blue LED
I have imported a SQL table that had 13 million records and 38 columns instantly through a JDBC connector.  Conversely, I am trying to load a SQL table that has 90 million records and 219 columns through the same connector but it seems like Paxata is spinning its wheels trying to load the table.  Is there a limitation on the table size that Paxata can import?  Is there a setting that can be applied in the application or on the server side to bring in this large table?

Labels (1)
0 Kudos
3 Replies
Highlighted
Linear Actuator
@CONCJA

There is a configured row limit for imported datasets named "Library Row Limit". This limit can be updated by your Paxata administrators, but defaults to 250m rows per dataset. 

When importing from databases, it is difficult to compare tables based purely on rows and columns. 

Using your numbers, the 90m row table accounts for roughly 40x the number of cells of the 13m row table.
13m rows * 38 columns = 494m cells
90m rows * 219 columns = 19.7b cells

We also have to consider the types and lengths of the columns. A table with several long char/varchar/clob columns will be far slower to import due to the larger number of bytes per row. 

Are you able to see the import progress advancing in the Status column on the Library List? 
Image: https://us.v-cdn.net/6030933/uploads/editor/ao/ctz7gaxg1x36.png 

Thanks for your post,
Bill
0 Kudos
Highlighted
Image Sensor
Hi CONCJA, there are some theoretical limits to the size of an import, but there are likely other factors at play here that are causing the wheel spinning. Assuming a similar density (average amount of data per cell), the second table you're importing is ~40x larger than the first and should take proportionately longer to import. Other factors like bandwidth availability, Paxata core server utilization, and data source utilization can all have impacts as well. Did the import ever finish or did it fail?

Diagnosing these issues can be difficult without looking closely at the logs of Paxata and sometimes the data source as well. If you'd like help looking into this import, please reach out to your Paxata Customer Success contact and we can take a look with you.

Thanks!
0 Kudos
Highlighted
Blue LED
Thank you for the replies. 

I did wait a little longer for the 90 million records to load and Paxata was able to load the data.  I believe it took over a half hour to load the data.  I am curious @JoshCKF  based on the factors you listed whether my organization's setup is at its optimal state to import such large datasets.  I may reach out to Paxata Customer Service to review my organization's setup.

Thanks.
0 Kudos