天道酬勤,学无止境

How to Set an existing ECS cluster to a compute environment in AWS Batch

Question

I have an ECS cluster running in my environment and it is configured with tasks, repositories and services and autoscaling.

I am discovering the AWS Batch service and when I create a new compute environment a new cluster is being added to the ECS service. I want to know how to set a compute environment to my already existing cluster?

Answer1

I know this is quite an old question and that it's not exactly the same case, but I was struggling with something similar, so it might be helpful to someone. In my case Batch was overwriting my ECS_DISABLE_IMAGE_CLEANUP setting, causing my special dedicated super-high-priority Job Queue (with its Compute Environment) lose its images, causing an annoying delay when they had to be pulled again.

To address this, I had to create a Launch Template (in EC2 console), which had everything set as "don't change", and some extra "user data" (at the end of the Advanced section):

 MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="==MYBOUNDARY=="

--==MYBOUNDARY==
Content-Type: text/x-shellscript; charset="us-ascii"

#!/bin/bash
echo ECS_DISABLE_IMAGE_CLEANUP=true>>/etc/ecs/ecs.config 

--==MYBOUNDARY==--

Then I had to recreate the Compute Environment as to use this Launch Template, which cause this extra line to be added to the ecs.config file and overwrite the Batch setting. Maybe it will work for the cluster, too!

Answer2

Per my knowledge (spent half of day, studying documentation and forums), that's impossible. computeEnvironments's ecsClusterArn is assigned immediately after creation of compute environment, and cannot be changed.

Interesting is that Batch somehow modifies /etc/ecs/ecs.config on newly created instances in managed CE:

... // config from my AMI ECS_CLUSTER=my_cluster_name ... // ECS_CLUSTER={ENVIRONMENT NAME HERE}_Batch_{RANDOM UUID HERE} ECS_DISABLE_IMAGE_CLEANUP=false ECS_ENGINE_TASK_CLEANUP_WAIT_DURATION=2m ECS_IMAGE_CLEANUP_INTERVAL=10m ECS_IMAGE_MINIMUM_CLEANUP_AGE=10m ECS_NUM_IMAGES_DELETE_PER_CYCLE=5 ECS_RESERVED_MEMORY=32 ECS_AVAILABLE_LOGGING_DRIVERS=["json-file","awslogs"] , and first ECS_CLUSTER is wiped with second. That looks like some predefined user-data, but I couldn't find where that can be changed either.

However, probably you can achieve original goal with reversed approach -- create unmanaged CE, and loop over instances you want to be in that CE, change ECS_CLUSTER in /etc/ecs/ecs.config to $that_autogenerated_CE_arn, restart ecs-agent (or relevant docker, if that's not Amazon Linux). Now your containers belong to another cluster, and only difference is in cluster's name.

Restricted HTML

  • Allowed HTML tags: <a href hreflang> <em> <strong> <cite> <blockquote cite> <code> <ul type> <ol start type> <li> <dl> <dt> <dd> <h2 id> <h3 id> <h4 id> <h5 id> <h6 id>
  • Lines and paragraphs break automatically.
  • Web page addresses and email addresses turn into links automatically.

相关推荐