Hi Srikanth,
To help clarify and note the differences and compliments, here's a solution integrating DataRobot text processing output, Azure key phrase extraction, and use of Power BI for data prep. The Azure key phrase extraction is going to identify the important terms, and Power BI is going to allow you to integrate those terms and then filter/slice based on other values in your dataset to create interesting visualizations. Key phrase extraction is going to give you the terms back returned from an API call. Power Bi is going to be able to digest it, present it, and you can merge it back with your original dataset.
Where DataRobot comes in is to add a layer of associating the of important terms with prediction results. So while Azure returns the important terms given some generalized corpus or context, DataRobot tells you the importance with respect to the specific thing you're trying to predict.
Now, you can get a lot more information about the terms than just the top few important ones. DataRobot shows you degree of importance as a strength each term contributes to the prediction made. In other words, a term might be important with respect to the frequency of occurrence or its specificity to a given context, but it may have no impact on the prediction results. Say we want to know user sentiment on movie reviews. The word movie might be important, but has no indication or relevance to sentiment. To get this data, you need to download it from one or more of our Word Cloud (under Models > Insights>), as linear model coefficients (Models > click on a linear model on the Leaderboard with a β𝑖 icon like Elastic-Net > click on Coefficients), similarly via coefficients show in Text Mining (under Models > Insights>), as a ratings table in a Generalized Additive2 Model, or as a 5th option which allows you to get the top most and least important word coefficients directly from the text modeler itself by similarly downloading coefficients from an "Auto-Tuned Word N-Gram Text Modeler" model. I encourage you to explore them each for their differences. You can get all terms and phrases, perhaps several thousand terms and phrases, that vary depending on which you use.
You can download each by clicking Export on the bottom right on most pages. This downloads as a csv file. You can also access this data from our API. Here's a link to the word cloud: https://datarobot-public-api-client.readthedocs-hosted.com/en/v2.22.1/autodoc/api_reference.html#wor.... Here for coefficients: https://datarobot-public-api-client.readthedocs-hosted.com/en/v2.22.1/entities/model.html#model-para..., or here in a ratings table: https://datarobot-public-api-client.readthedocs-hosted.com/en/v2.22.1/autodoc/api_reference.html#dat.... You would then import this data into Power BI to aggregate it all together.
I have a few thoughts to integrate this term strength and the prediction results back to the source data. The predictions occur one to one to your source - you can simply add the column so that's straight forward. The term strength would be mapped to the key phrase extraction importance with a little data munging. Look up each term in the csv you downloaded from DataRobot and create new columns with the DataRobot strength. Or you could replace the Azure terms with their strength. Or you could group the DataRobot terms strengths into a group of say 1 to 10 and map that encoding back to the dataset. You could them re-run your new dataset back through DataRobot to get even more refined predictions. There are lots of creative options. Or you could not use Azure's extraction importance and you can instead use DataRobot's prediction based importance, though both offer you interesting insights from differing qualities of 'importance'. In any case, you now have the Azure term importance, the DataRobot predictions, the DataRobot term prediction target relationship, and your source file all aggregated in Power BI.
As far as your Hadoop integration, please have a look at this: https://community.datarobot.com/t5/resources/deploying-a-model-to-hadoop/ta-p/5044. And here for data connectors in general: https://community.datarobot.com/t5/resources/importing-data-overview/ta-p/1712. We can connect to any data source in a variety ways - through JDBC, with specific vendor connectors via our connectors to them or theirs to us, with static and dynamic SQL, or just an API if you want to put some application code in the middle. When it comes time to think about what's best for your application (low latency, cheapest overhead, etc), you can assess all options when you're ready. But my personal take during development is to do what's easiest to prove out this integration first, which for me is to code it up and connect via the API.
I hope this helps!
日本語
