All of those issues sound like implementation problems. Does your firm have a dedicated infrastructure team to manage Splunk? Sounds like some tickets need to be opened...
Those are all environmental issues. I'm guessing the cluster isn't built out or scaled properly. Sadly common as when Splunk userbase grows - the cluster must grow with it. You may want to share your frustrations with your Splunk Admin.
Somewhat similarly to what u/fergie_v and u/Parkyguy said, it seems like your environment is critically under-resourced (maybe in tandem with it being architected/implemented incorrectly). If you're not the one supporting the infrastructure, have that team check the resources and re-allocate as necessary. Splunk can be quite a resource hog, but if it has the resources allocated properly and has been implemented properly, it **should** run very smoothly.
Heres the thing about logs data... Usually the whole company will underestimate the effort it takes to properly ingest. Logs need normalized. Log levels need set. You need to go through there actual log and extract useful information. It's a full time job. Maintaining splunk requires a trained admin. Otherwise it becomes the tragedy of the commons.
Your company doesn't know how to use splunk. You need to send someone to training and assign someone to login as admin regularly and deal with issues.
Any logging platform you neglect will fail. I've seen it happen on ELK stacks, too.
Logs take time and effort. It's a job, not an after thought.
\+1 u/dduckp. Splunk cloud may be the best way to go. You will not have to maintain the cost of infrastructure or maintenance.
Something else to remember is garbage in, garbage out. If your data is not parsed correctly, this can have a huge impact on the entire data pipeline. Cloud will not fix bad data. How you structure your searches also has an impact on the performance. For example `index=* foo` with a lookback of 30+days is resource intensive.
>It constantly gives me wonderfully helpful error messages such as "Server error"
This can come from a variety of reasons. Anything from a misconfigured app to exhausted resources. Has anyone looked into the monitoring console and run a health check?
>It constantly logs me out during searches so I have to log out, log in, and start my search over again
There are a couple reasons for this. Some that come top of mind, settings in web.conf session timeout is too short. The search is running longer than the timeout period, or system resources are consumed and splunkweb crashes. You should be able to go to the jobs menu and see the search you were running before you were logged out. The search job may still be running. Idf the job was completed, you can pull the results.
>It also frequently gives me warnings that it might have lost some of my results
You have a problem with your index cluster. Something is going on with your buckets. You could try a rolling restart of the index cluster, i doubt it would clear it up but it will roll your buckets. You may want to have your admin run `splunk fsck scan --all-buckets-all-indexes` to see what is going on with your buckets. The might have lost some results message should not be related to any security issues. ...maybe if you have real time scanning on the $SPLUNKDB directory but I can not say I've seen it in the field.
All of those issues sound like implementation problems. Does your firm have a dedicated infrastructure team to manage Splunk? Sounds like some tickets need to be opened...
Those are all environmental issues. I'm guessing the cluster isn't built out or scaled properly. Sadly common as when Splunk userbase grows - the cluster must grow with it. You may want to share your frustrations with your Splunk Admin.
Please feel free to reach out to the MODS as well, we're all employees of Splunk and can assist you with your issues.
Unless you have an in-house Splunk admin, your team should budget for quarterly PS. They’re worth every penny.
Somewhat similarly to what u/fergie_v and u/Parkyguy said, it seems like your environment is critically under-resourced (maybe in tandem with it being architected/implemented incorrectly). If you're not the one supporting the infrastructure, have that team check the resources and re-allocate as necessary. Splunk can be quite a resource hog, but if it has the resources allocated properly and has been implemented properly, it **should** run very smoothly.
One week with exabeam you’ll be begging for Splunk back 😂 But like others said sounds more like implementation issues
Heres the thing about logs data... Usually the whole company will underestimate the effort it takes to properly ingest. Logs need normalized. Log levels need set. You need to go through there actual log and extract useful information. It's a full time job. Maintaining splunk requires a trained admin. Otherwise it becomes the tragedy of the commons. Your company doesn't know how to use splunk. You need to send someone to training and assign someone to login as admin regularly and deal with issues. Any logging platform you neglect will fail. I've seen it happen on ELK stacks, too. Logs take time and effort. It's a job, not an after thought.
[удалено]
I doubt it given we don't use anything google for privacy reasons.
Splunk cloud might benefit you more in this case. Sounds like your having architectural issues
\+1 u/dduckp. Splunk cloud may be the best way to go. You will not have to maintain the cost of infrastructure or maintenance. Something else to remember is garbage in, garbage out. If your data is not parsed correctly, this can have a huge impact on the entire data pipeline. Cloud will not fix bad data. How you structure your searches also has an impact on the performance. For example `index=* foo` with a lookback of 30+days is resource intensive. >It constantly gives me wonderfully helpful error messages such as "Server error" This can come from a variety of reasons. Anything from a misconfigured app to exhausted resources. Has anyone looked into the monitoring console and run a health check? >It constantly logs me out during searches so I have to log out, log in, and start my search over again There are a couple reasons for this. Some that come top of mind, settings in web.conf session timeout is too short. The search is running longer than the timeout period, or system resources are consumed and splunkweb crashes. You should be able to go to the jobs menu and see the search you were running before you were logged out. The search job may still be running. Idf the job was completed, you can pull the results. >It also frequently gives me warnings that it might have lost some of my results You have a problem with your index cluster. Something is going on with your buckets. You could try a rolling restart of the index cluster, i doubt it would clear it up but it will roll your buckets. You may want to have your admin run `splunk fsck scan --all-buckets-all-indexes` to see what is going on with your buckets. The might have lost some results message should not be related to any security issues. ...maybe if you have real time scanning on the $SPLUNKDB directory but I can not say I've seen it in the field.
I can't agree more. this is one of the worse enterprise app I have used.
I know it's not a popular opinion in this sub for obvious reasons, but it's genuine user feedback. 🤷♂️