Incremental Refresh / Appending / Stacking Data in Paxata
Quick question. I was wondering if Paxata has the capability to perform an incremental refresh. I have a set of excel files I receive every month and would like to stack it on a monthly basis. This would be considered an incremental refresh. Unfortunately, I haven't figured out a way to complete this task in Paxata.
It doesn't seem paxata supports this. I tried creating a standard excel file then adding a version, but that didn't work. Then tried creating a foundation data set then created another data set which was set to automate (new data) and then appending that on the foundational data set within a project hoping that project would keep all the appended data. However, no dice there as well.
Hello, I may need a bit of clarification in order to answer your question - are you importing the Excel files locally or from a shared area like Sharepoint? Paxata does not support the automation of local file imports into our Library (don't want the server reaching out to individual desktops) but we do support automated import from enterprise sources (SFTP, Sharepoint, S3, WASB, databases, etc.). If you load the latest version of the Excel file into the Library, either on demand if a local file or on a schedule if enterprise system then you can create a project which starts with the original file and Appends the updated file. When there is a new updated file in the Library you will notice a "refresh datasets" button at the bottom of the Steps panel in the project turns green. You can select this option and then choose to update the latest version of any new datasets in the project. You can also set automation options to use "latest version". Does this help? Please let me know if I can provide additional clarification.
No, it is not possible to automate the import of local files from your desktop into the Paxata Library. If you click on a dataset in the Library you do have the option to add a version - this will help simplify the Library so that you can view each version of a dataset vs. adding new versions as a separate/distinct dataset in the Library.
Import can be based on either the entire table or a query. If using a query then you could incorporate a condition in the where clause based on current day or some other flag that would capture updated rows. You can then automate this to run. I hope this helps!
Hey Martha, we came up with a question related to the start of this thread. Is there any documentation that explains how to incrementally append data to a dataset? This becomes circular at some point and we're having trouble figuring it out. It seems like you would have to be able to modify a current dataset with a project without having the project create a new dataset?