The year started with the brilliance of my work and I was on high when everything changed in my life. I did face a few challenges to overcome and pushed into the job that made me learn almost every technology from being a developer, I taught to think like a DevOps guy, Consultant, Architect, and a Manager, I did few mistakes and my situation was all over me half of the year but It never made me forget my root — Innovation and Problem-Solving.
In this article, I wish to list out a few of the challenges that I faced and successfully solved them in the customer place. This is my professional journey of 2019. It is like a pill in Ibiza. Being a developer, I indulged to help customers as a consultant. My confession of the whole year is that If I go back and do things again, “I would love to do it more differently and dedicatedly.”
Honorable mentions, HPCL, IndusInd Bank and Zycus. Let’s take a look at the different challenges I faced,
List of Challenges:
- Dynatrace Mobile Application Configuration for Managed Installation Clusters
- Dynatrace Mobile Instrumentation for Kony Platform
- Network Metrics in Dynatrace Mobile Monitoring
- User Session Retention Period
- Node Balancing:
- Elasticsearch Streaming:
- API Reporting Using Dynatrace APIs:
- Synthetic Monitoring with Captcha (Proscript):
- RUM Automatic Maintenance:
- Kubernetes Offline Installation:
- Xamarin Mobile Instrumentation:
- Cordova Application Instrumentation:
- Solaris System Monitoring:
- URL Monitoring without Consuming DEM Licenses
Dynatrace Mobile Application Configuration for Managed Installation Clusters
When we implement Dynatrace in a customer place, we often forget to make Cluster Active Gate public unless we require it. It is recommended to make AG (Active Gate) public that is because it has its pros. It helps to monitor websites and check their availability around the world and the main thing is to get problem notification on the go. We need to make cluster AG public to monitor our mobile apps as well.
The challenge was, even cluster AG is public, customers were not able to get problem notifications to their mobile. There were 2 options in front of me to decide, I can integrate Slack to get notifications or I can find out the root cause and do it. I tried the first one and failed. Customers had partners who were taking care of their APM solution. What I had to do should be gone through partners. When I tried to Implement Slack, they laughed at me and made slack workspace fail. Hence, customers never adopted it. I only left with one choice now, i.e. to find the root cause and help them to get the notifications in Dynatrace Mobile App.
Dynatrace cluster is not accessible to outside their premises that was the root cause. I tried an idea that came to me as I have some experience about how mobile apps work to connect to the server hence, I downloaded the application and to connect their cluster, I connected to the customer’s WiFi network where Dynatrace cluster was also accessible and tried connecting mobile app to it. It got connected to their cluster. Now the problem is that the application is built to get notifications on the go. When a person goes out of the office, he will not be able to get problem notifications. But, guess what, he got notifications on the go! This is because, the application was getting connected at the start using beacon URL from the browser, when the network the browser is in, cannot access the cluster, the application was failing to connect to it. Upon first connection, the application receives the data through cluster AG that is made public.
Dynatrace Mobile Instrumentation for Kony Platform
As we all know in 2018, Mobile Instrumentation was just growing in Dynatrace. We were not supported officially. I worked on it to support it with conventional methods. Just instrumented it with the offline plugin and voila! Back to business. It got instrumented. this was the first production mobile application I instrumented and from that day I have helped to instrument around 25 mobile apps this year.
Network Metrics in Dynatrace Mobile Monitoring:
This is where I used my head more. The customer was happy with Dynatrace and they expected all the features of web application monitoring to be there in mobile monitoring as well. These customers knew AppMon and they needed to see the same kind of data in Dynatrace. I was new and was getting started to know about Dynatrace but I had my skills in Android development from the past 3 years.
I found out a workaround to convince them and at last they did not accept due to huge commercial involved. The problem was that they were not getting any data about Front-End Time, Network Time and Server Time. Mobile Instrumentation cannot detect those times as we were able to capture only front end time and web request timing in it. Here in this crisis, I found our Hybrid Instrumentation. It is better than auto-instrumentation and helps to have control over monitoring data. Hybrid Instrumentation is the combination of auto and manual instrumentation without affecting the auto instrumentation procedures.
User Session Retention Period
In Dynatrace, we can store the data of User sessions for only 35 days and thereafter, all the data will be truncated. To address this issue, I contacted support and they told me. “this is because monitoring made simple and storing user session data for longer than it is required is not a good idea. It just simply stacks up unnecessary information.” One way or the other it made sense to me.
But, in our customer places, they need to store the data for more than 35 days for example, in Bank sectors. There is a solution I found out to stream data to Elasticsearch and store all the data in the cluster. There is a limitation in Elasticsearch as well. There is a premium subscription required to store unlimited data and to process it.
To make this thing easy, I found out a workaround where we can have our own MongoDB instance setup in customer place and then we can store all the user session data in the database. This can be done through Dynatrace API. I created an API call to retrieve all the user sessions with the user actions for the given amount of time and fetch the data and store that data in MongoDB or Firebase. About this data, I have created a portal that can plot charts in a portal with the data for future analysis. The data retention period of this portal was unlimited. Hence, this solved the problem of data retention.
Node Balancing
In a Managed installation of Dynatrace, we recommend our customers to have at least 3 nodes to store the data including, transactional storage and non-transactional storage. Dynatrace runs even when there is one node. There was a situation in the customer place that they asked the reason, why dynatrace is recommending 3 nodes instead of 2?
Everybody suggested that to make sure there is a double safety upon cluster failover, Dynatrace recommends 3 nodes. When I searched for the answers with my team members, I have found out an interesting fact about it. We thought before that Data stored in all those 3 nodes are redundant and also stored as a backup to each other. The fact is, Dynatrace is storing dividing the data that it is storing in nodes equally. For example, if there are 3 user sessions, all 3 of them will not be stored in all 3 nodes. Instead, each session will be stored in each of the nodes.
Data is not a replica in other nodes. If one cluster fails, that means Dynatrace will have only 66% of your data and rest 33% will not be stored in nodes Hence, you need to set up a backup to store complete data. Your setup can have a maximum of 8 nodes depending on your requirement. The cluster can take up to a maximum of 3 failure nodes. More than 3 failovers can lead to the whole setup failure.
Elasticsearch Streaming
In our product, to stream user session data away from Dynatrace we have an option in settings to stream user sessions directly to Elasticsearch instance with the index value. The limitation of the feature is that you cannot stream data from multiple management zones. You can either stream the data of one management zone or all management zones.
Consider the situation. Elasticsearch instance is down due to some maintenance error for 30mins of time. Dynatrace continuously streams the user session data to Elasticsearch. At this moment, if Elasticsearch is down, the customer loses the data until Elasticsearch gets back running.
Currently, it is a continuous stream of documents with only a few retries before the exported document is discarded. So for a short interruption document will be retried after a short while and sent when it is available again, but if Elasticsearch and offline for a longer time, documents are discarded at some point with an admin-visible notification in the WebUI. The retry count is 3.
To tackle this issue, I built a portal similar to that of the User session retention period. In here, I call Dynatrace API to fetch all the user session data of the last 24 hours every day and store it in MongoDB or Firebase (Any JSON Database). This data I can feed it to Elasticsearch or my portal. The main thing is data is not lost.
API Reporting Using Dynatrace APIs
I call this workaround as the best one I ever did till now. Dynatrace reporting is amazing and it gives us the complete health report and it is generated automatically every week. In that report, all the data are shown as scores by calculating them as Application Score, Service Score, and Infrastructure Scores. In customer places, they needed more info like what is the average Host health, Number of sessions captured, Details of the problems that have been reported by Dynatrace among how many are service-based or infrastructure based? Which of them has the root cause? These are the concerns that they had.
To make Dynatrace better in those cases as well, I built a portal that contains reporting patterns based on the data sent by the customer like, If they provide their API key, Tenant details, Required metrics, Timeframe and the Scheduler duration the portal will directly start reporting to the customers with their specified metrics in the given time of choosing timeframe. Currently, it uses an excel format to send reports. In the future, it will have more capabilities like Business Intelligence and Visualizing reports along with an Issue tracker.
The development of the API reporting platform uses the Dynatrace API feature to receive data from the tenant. Every step is taken to ensure the safety of the data and it won’t be compromised to any third-party servers without a security check.
Synthetic Monitoring with Captcha (Postscript)
Synthetic Monitoring is one more feature in Dynatrace that always catches my eye. It is used to monitor URLs and Browser clickpath that monitors the availability of the URL in different locations all around the world and also we can use this in our own custom locations to monitor as well.
While using clickpath, if we encounter a login page and if there is any captcha verification on that page, the synthetic monitor stops right there and it won’t proceed further. To make this work we need to use Prescript and Postscript. We need to add an access token and this will do the job for us. In this case, the support team has helped us. This is a sensitive workaround therefore, only privileged members who have access can do this job.
RUM Automatic Maintenance
This one was the smart one for me. This uses the Dynatrace Configuration APIs. We don’t have an option in Dynatrace to schedule Real User Monitoring (RUM) to the apps. For example, if we want to turn off monitoring of the application in Dynatrace after business hours and resume it when business hours start, we need to manually do it every day and if we miss to turn it off, we lose licenses and if we forget to turn it on, we lose business data from the users. If we keep 24/7 enabled, there might be chances to consume licenses for unnecessary data.
My customer was worried about the process and they lost too many licenses and the data because of the human error. To minimize this, I did a simple workaround. I used Postman’s Monitor and Dynatrace Config APIs. I collected their RUM maintenance APIs, created 2 collections in Postman and added to monitor. One is to call an API with isrumenabled parameter to true and others to false. This sends a call to that particular web application to turn on/off RUM in the given time. Postman’s Monitor has that feature to set time and intervals to execute the API call.
Kubernetes Offline Installation
This is the recent query I got to answer. I had no idea about Kubernetes and how it works. Apart from knowing the term, I knew nothing about it. This assignment has been passed to me by one of my friends. I opted to help her and to work on it.
Installation of OneAgent operator in Kubernetes is a direct approach as per Dynatrace Documentation. As we all know ideal conditions are far away when we are in customer places. In their cluster, there is no internet. Without the internet, a direct approach is not possible because it connects to the red hat server to download the operator. To tackle this issue, I asked our support team. They insisted on a workaround to host the operator in their local repo and then pointing out to that location in the JSON file obtained in Dynatrace’s GitHub repo. So, download the operator, hosted in the local repo then pointed the location to that repository and later executed commands based that.
Challenge was hosting the operator in the local repo. We needed to tag the local repo with remote repo and then pushing all those files to the local repository. This task seems easy for experts but for a person like me without having knowledge and later after a week to pull this task off is a kind of satisfactory work.
Xamarin Mobile Instrumentation
This was a unique challenge for me. When Xamarin was launched, I was working as a student developer at Microsoft. I loved each and every release of Visual Studio and was obsessed with their technology. I explored Xamarin in its early stages and developed an app to learn Xamarin.
That really came in handy for Mobile Instrumentation. There was a project came up to instrument the Xamarin app. Nobody was there in my team to assist them with that. As my expertise exists in mobile applications, I was insisted to work on this small task of Instrumenta=ing Xamarin apps. When I learned to instrument these apps by myself, I found out that it is similar to iOS instrumentation. Need to add dynatrace.properties file to the root directory and rest is bug fixing and finding compatible third-party libraries.
Cordova Hybrid Application Instrumentation
The most challenging instrumentation I have ever faced this year. Our many of bigger clients are using Cordova. I heard about Cordova when I attended Tizen Dev Lab conducted by Samsung. I met a developer who has around 15 years of experience in the industry talking about cross-platform application development tools.
When I asked to take up the Cordova instrumentation, I had no idea how it works internally. I know only Android Instrumentation that too because I worked only on Android. I choose to ask our product team about it. After creating the sample application of Cordova in my local machine, I began instrumenting the application. I downloaded dynatrace.config from my Dynatrace tenant and added that file to the root directory of my project. The sample application was Instrumented but there was only native code was instrumented. Js code that is there in the project is not instrumented. So, I have created a web application and injected an agent to it. This instrumented web requests but not native requests.
There was confusion there. How to get bot the requests in a single application? Later I found out that while generating .config file from the tenant, we need to enable web monitoring and this injects the Js agent to the file. With the help of the config file, Cordova downloads Js agent from the tenant specified. This will instrument both native and web parts together. Still, we need to see the web requests in the web application version of the mobile app in Dynatrace tenant.
React Native Instrumentation
React Native Instrumentation is now official in Dynatrace but before 4 months it wasn’t. We had to use an unofficial version of the plugin found in GitHub. Recently, a customer with the React Native app came across and asked me to help them with the instrumentation. In the documentation of Instrumentation, there is no clear or direct information about how to create dynatrace.config file using properties. So, they used the same config file that is generated by Cordova Instrumentation. React Native config file is different from that file hence, I have created the config file of my own and supported them with that. Finally, Instrumentation was successful.
Instrumenting Mobile applications that have Keyboard
There is an International customer who builds a Keyboard for customers and that helps them to check their account balance right from the keyboard without opening their banking application. For these kinds of applications, if we do auto-instrumentation, it captures everything that the customer touch. This will become a key logger. To eliminate the possibility, we had to find a workaround. Manual instrumentation is not the best option because auto instrumentation benefits will not be there if we go with the manual.
There will be a lot of code changes to make and this is not the feasible way. Thus, we found a way to start the agent wherever the particular activity loads in the app. Upon closing that activity, Plugin stops capturing the data and send it to Dynatrace. So, stopped autostart of agent whenever application loads and started the agent whenever they need. Even this worked out for certain key clicks as well.
Solaris System Monitoring & URL Monitoring without Consuming DEM Licenses are the upcoming challenges that I am working on for the year 2020. These workarounds are demanding the best out of me and it makes me curious to know the product better.
Conclusion:
My mind is always out of the box and I never accustomed to working on conventional methods to resolve the problem. Due to my style os work, I took the hit from my top-level individuals and got a complaint that I am not fit to work and I am irresponsible. It is not how much we are disciplined to deliver the service. It is about how best we can deliver the service to the customer that makes them think more than conventional ways.
When Dynatrace itself ranked as a leader in innovations, why to follow conventional methods to stay in the race? we can be innovative as well as the whole company is. That is where we stand
Originally published at https://www.linkedin.com.
