I’ve run another test where I start the elasticsearch-? without kibana.
I must be missing something simple. Should we arrange a walkthrough?
The errors are:
elasticsearch-2 | {"@timestamp":"2022-10-17T23:47:07.103Z", "log.level":"ERROR", "message":"error downloading geoip database [GeoLite2-ASN.mmdb]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es02][generic][T#3]","log.logger":"org.elasticsearch.ingest.geoip.GeoIpDownloader","elasticsearch.cluster.uuid":"I8OLmKzIR0-zFhk4DYAq0A","elasticsearch.node.id":"bixZpHrqTaWAVnr9KyJiLA","elasticsearch.node.name":"es02","elasticsearch.cluster.name":"elasticsearch-cogstack-cluster","error.type":"org.elasticsearch.action.UnavailableShardsException","error.message":"[.geoip_databases][0] primary shard is not active Timeout: [1m], request: [BulkShardRequest [[.geoip_databases][0]] containing [index {[.geoip_databases][GeoLite2-ASN.mmdb_0_1666050336214], source[n/a, actual length: [1mb], max length: 2kb]}]]","error.stack_trace":"org.elasticsearch.action.UnavailableShardsException: [.geoip_databases][0] primary shard is not active Timeout: [1m], request: [BulkShardRequest [[.geoip_databases][0]] containing [index {[.geoip_databases][GeoLite2-ASN.mmdb_0_1666050336214], source[n/a, actual length: [1mb], max length: 2kb]}]]\n\tat org.elasticsearch.server@8.3.3/org.elasticsearch.action.support.replication.TransportReplicationAction$ReroutePhase.retryBecauseUnavailable(TransportReplicationAction.java:1074)\n\tat org.elasticsearch.server@8.3.3/org.elasticsearch.action.support.replication.TransportReplicationAction$ReroutePhase.doRun(TransportReplicationAction.java:874)\n\tat org.elasticsearch.server@8.3.3/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)\n\tat org.elasticsearch.server@8.3.3/org.elasticsearch.action.support.replication.TransportReplicationAction$ReroutePhase$2.onTimeout(TransportReplicationAction.java:1033)\n\tat org.elasticsearch.server@8.3.3/org.elasticsearch.cluster.ClusterStateObserver$ContextPreservingListener.onTimeout(ClusterStateObserver.java:345)\n\tat org.elasticsearch.server@8.3.3/org.elasticsearch.cluster.ClusterStateObserver$ObserverClusterStateListener.onTimeout(ClusterStateObserver.java:263)\n\tat org.elasticsearch.server@8.3.3/org.elasticsearch.cluster.service.ClusterApplierService$NotifyTimeout.run(ClusterApplierService.java:649)\n\tat org.elasticsearch.server@8.3.3/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:710)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)\n\tat java.base/java.lang.Thread.run(Thread.java:833)\n"}
That repeats
Then I fire up kibana, and see the following errors:
cogstack-kibana | [2022-10-17T23:49:35.342+00:00][ERROR][elasticsearch-service] Unable to retrieve version information from Elasticsearch nodes. security_exception: [security_exception] Reason: unable to authenticate user [kibanaserver] for REST request [/_nodes?filter_path=nodes.*.version%2Cnodes.*.http.publish_address%2Cnodes.*.ip]
Credential setting gives the same error:
ELASTIC_HOST not set, defaulting to ELASTIC_HOST=localhost
ELASTIC_PASSWORD not set, defaulting to ELASTIC_PASSWORD=kibanaserver
ELASTIC_USER not set, defaulting to ELASTIC_USER=elastic
KIBANA_USER not set, defaulting to KIBANA_USER=kibanaserver
KIBANA_PASSWORD not set, defaulting to KIBANA_PASSWORD=kibanaserver
INGEST_SERVICE_USER not set, defaulting to INGEST_SERVICE_USER=ingest_service
INGEST_SERVICE_PASSWORD not set, defaulting to INGEST_SERVICE_PASSWORD=ingest_service
Waiting for Elasticsearch availability
curl: (6) Could not resolve host:
{
"name" : "es01",
"cluster_name" : "elasticsearch-cogstack-cluster",
"cluster_uuid" : "I8OLmKzIR0-zFhk4DYAq0A",
"version" : {
"number" : "8.3.3",
"build_flavor" : "default",
"build_type" : "docker",
"build_hash" : "801fed82df74dbe537f89b71b098ccaff88d2c56",
"build_date" : "2022-07-23T19:30:09.227964828Z",
"build_snapshot" : false,
"lucene_version" : "9.2.0",
"minimum_wire_compatibility_version" : "7.17.0",
"minimum_index_compatibility_version" : "7.0.0"
},
"tagline" : "You Know, for Search"
}
Setting kibana_system password
{"error":{"root_cause":[{"type":"unavailable_shards_exception","reason":"[.security-7][0] [1] shardIt, [0] active : Timeout waiting for [1m], request: indices:data/write/update"}],"type":"unavailable_shards_exception","reason":"[.security-7][0] [1] shardIt, [0] active : Timeout waiting for [1m], request: indices:data/write/update"},"status":503}Creating users
{
"error" : {
"root_cause" : [
{
"type" : "unavailable_shards_exception",
"reason" : "[.security-7][0] primary shard is not active Timeout: [1m], request: [BulkShardRequest [[.security-7][0]] containing [index {[.security][user-kibanaserver], source[{\"username\":\"kibanaserver\",\"password\":\"$2a$10$4midskHtrgJm6jr8YAEd/.SFmhRDTXpOrHWTXsN/LYpKeaG71G1hW\",\"roles\":[\"kibana_system\",\"kibana_admin\",\"ingest_admin\"],\"full_name\":\"kibanaserver\",\"email\":\"cogstack@admin.net\",\"metadata\":null,\"enabled\":true,\"type\":\"user\"}]}] and a refresh]"
}
],
"type" : "unavailable_shards_exception",
"reason" : "[.security-7][0] primary shard is not active Timeout: [1m], request: [BulkShardRequest [[.security-7][0]] containing [index {[.security][user-kibanaserver], source[{\"username\":\"kibanaserver\",\"password\":\"$2a$10$4midskHtrgJm6jr8YAEd/.SFmhRDTXpOrHWTXsN/LYpKeaG71G1hW\",\"roles\":[\"kibana_system\",\"kibana_admin\",\"ingest_admin\"],\"full_name\":\"kibanaserver\",\"email\":\"cogstack@admin.net\",\"metadata\":null,\"enabled\":true,\"type\":\"user\"}]}] and a refresh]"
},
"status" : 503
}