This April, Hortonworks launched a multi-phase initiative to streamline Apache Hadoop operations, and the 1.3 release of SmartSense marks the delivery of the second phase of that initiative, and that is to provide Consolidated Cluster Activity Reporting.
Hortonworks launched SmartSense in 2015 to help customers quickly collect cluster configuration, metrics, and logs to proactively detect issues, and expedite support case resolution. By combining the analytical capabilities of SmartSense, with the interactive reporting capabilities of Apache Zeppelin, SmartSense 1.3 now provides consolidating utilization data reporting across core HDP components in a single dynamic UI available on-premise.
SmartSense mines, consolidates, and stores YARN, MapReduce, Tez, and HDFS utilization data and provides a fully managed Apache Zeppelin instance with pre-built notebooks to analyze, query, and visualize that data. Out of the box notebooks can be extended, and custom notebooks can be created to support customer-specific reporting needs.
This solution allows customers to easily and quickly answer questions like:
- What are the top ten most active users?
- Which queues consume the most CPU resources?
- Which jobs take the longest and consume the most memory on my cluster?
- Which jobs read and write the most HDFS data?
- Who’s creating all my small files?
- How much should I charge my users for their resource utilization?
We believe this capability will be a game changer for our customers and give them the data they need to make better business decisions, and more effectively understand, troubleshoot and manage their cluster.
Simplified Bundle Upload Options (SFTP and HTTPS)
For many SmartSense users, outbound internet access from the HST server is not readily available. For those users, the HST Gateway has been created to simplify the process of uploading bundles of diagnostic information to Hortonworks Support. In SmartSense 1.3, users have more upload options with SFTP, HTTPS and authenticated HTTPS proxy support.
On top of that, we’ve embedded an instance of the SmartSense Gateway into the HST Server, so for clusters that have outbound internet access, no additional Gateway setup is required.
Expanded Anonymization and Redaction Capabilities
Giving customers the flexibility to pre-process cluster diagnostic information before it is encrypted and sent to Hortonworks has always been a priority in SmartSense. All bundles are encrypted and anonymized by default and in SmartSense 1.3 we’ve expanded the number of out-of-the-box Anonymization rules, giving customer more flexibility with how cluster diagnostic information is secured.
We hope you’re as excited about SmartSense 1.3.0 as we are, and here are a few links to get started: