天道酬勤,学无止境

Cannot catch exception in spring batch's ItemWriter

I'm writing a Spring Batch process to migrate a dataset from one system into another. In this case this is as simple as using a RowMapper implementation to build the object from a query before handing off to the ItemWriter. The ItemWriter calls the save method on my DAO (defined as an interface and handled by spring data)

The issue is this: I have a unique constraint on the MyItem table, and therefore saving a duplicated record will result in a DataIntegrityViolationException. I have tried catching this within the ItemWriter to allow me to log that a record was not imported, however during execution it never enters this catch statement. I have attempted catching Exception AND Throwable to no avail too.

From what I've noticed, there is a @Transactional annotation on the 'save' method of my DAO where I would expect the commit and flush to occur. Does Spring Batch alter this transaction in any way? Such that the @Transactional annotation applies to the 'write' method of the ItemWriter?

Can I even catch the exception in this class?

I've provided code snippets below, if you need further information - please let me know.

Many thanks for any help you can provide

ItemWriter

@Component
public class MyItemWriter implements ItemWriter<MyItem> {

    private static final Logger LOG = LoggerFactory.getLogger(MyItemWriter.class);

    @Resource
    private MyItemDao myItemDao;

    @Override
    public void write(List<? extends MyItem> myItems) throws Exception {
        for (MyItem myItem : myItems) {
            try {
                myItemDao.save(myItem);
            } catch (Throwable ex) {
                LOG.warn("Failed to import MyItem: {}: {} ", myItem.getId(), ex.toString());
            }
        }
    }
}

DAO

public interface MyItemDao extends PagingAndSortingRepository<MyItem, Integer> {
    [Custom methods omitted]
}

Spring Batch configuration

<batch:job id="myImportJob" restartable="true" job-repository="jobRepository">
    <batch:step id="myImportStep" allow-start-if-complete="true">
        <batch:tasklet>
            <batch:chunk reader="myItemReader" writer="myItemWriter" commit-interval="50" />
        </batch:tasklet>
    </batch:step>
</batch:job>

评论

A couple points here:

  1. Remove the @Transactional annotation on your DAO. @Transactional and Spring Batch typically don't play nice. Spring Batch manages the transactions as part of the framework's functionality and attempting to manipulate that functionality can cause unexpected side effects.
  2. As M. Deinum points out, your ItemWriter and therefore your DAO is participating in a transaction that Spring Batch is managing. Because of this, you won't get that exception until the transaction commits.

With the above considerations in place, you have two options:

  1. You can configure skip logic to skip records that throw that exception. If you need to log the item, you can add a SkipListener to the mix so that you can log the item that caused the exception. However, there is a performance penalty to be paid with this approach since throwing the exception will result in the transaction being rolled back and replayed one item at a time.
  2. You can filter the items via an ItemProcessor. This saves the performance hit for skip logic.

You can read more about Spring Batch's skip logic in section 5.1.5 here: http://docs.spring.io/spring-batch/trunk/reference/html/configureStep.html

受限制的 HTML

  • 允许的HTML标签:<a href hreflang> <em> <strong> <cite> <blockquote cite> <code> <ul type> <ol start type> <li> <dl> <dt> <dd> <h2 id> <h3 id> <h4 id> <h5 id> <h6 id>
  • 自动断行和分段。
  • 网页和电子邮件地址自动转换为链接。

相关推荐
  • 在Spring Batch中如何实现跳过?(How is the skipping implemented in Spring Batch?)
    问题 我想知道如何在ItemWriter中确定Spring Batch当前处于块处理模式还是回退单项处理模式。 首先,我没有找到有关该后备机制如何实现的信息。 即使我还没有找到解决我实际问题的解决方案,我也想与您分享我关于后备机制的知识。 如果我错过了任何事情,请随时添加其他信息的答案;-) 回答1 可以在FaultTolerantChunkProcessor和RetryTemplate中找到跳过机制的实现。 假设您配置了可跳过的异常,但没有配置可重试的异常。 并且当前块中有一个失败的项目导致异常。 现在,首先将整个块写入。 在处理器的write()方法中,您可以看到调用了RetryTemplate 。 它还获得了对RetryCallback和RecoveryCallback两个引用。 切换到RetryTemplate 。 查找以下方法: protected <T> T doExecute(RetryCallback<T> retryCallback, RecoveryCallback<T> recoveryCallback, RetryState state) 在那里,您可以看到RetryTemplate只要没有用尽RetryTemplate重试(即在我们的配置中恰好一次)。 这样的重试将由可重试的异常引起。 不可重试的异常将在此处立即中止重试机制。 重试用尽或中止后
  • ItemWriter的Spring Batch跳过异常(Spring Batch skip exception for ItemWriter)
    问题 我正在尝试将Spring Batch 2.2.5与Java配置一起使用。 这是我的配置: @Configuration @EnableBatchProcessing public class JobConfiguration { @Autowired private JobBuilderFactory jobBuilder; @Autowired private StepBuilderFactory stepBuilder; @Bean @Autowired public Job processDocumentsJob() { return jobBuilder.get("processDocumentsJob") .start(procesingStep()) .build(); } @Bean @Autowired public Step procesingStep() { CompositeItemProcessor<File, DocumentPackageFileMetadata> compositeProcessor = new CompositeItemProcessor<File, DocumentPackageFileMetadata>(); compositeProcessor.setDelegates(Lists.newArrayList(
  • Spring Batch skip exception for ItemWriter
    I'm trying to use Spring Batch 2.2.5 with Java config. Here is the config that I have: @Configuration @EnableBatchProcessing public class JobConfiguration { @Autowired private JobBuilderFactory jobBuilder; @Autowired private StepBuilderFactory stepBuilder; @Bean @Autowired public Job processDocumentsJob() { return jobBuilder.get("processDocumentsJob") .start(procesingStep()) .build(); } @Bean @Autowired public Step procesingStep() { CompositeItemProcessor<File, DocumentPackageFileMetadata> compositeProcessor = new CompositeItemProcessor<File, DocumentPackageFileMetadata>(); compositeProcessor
  • 如何在没有项目编写器的情况下编写春季批处理步骤(How to write a spring batch step without an itemwriter)
    问题 我正在尝试使用以下配置在没有项目编写器的情况下配置春季批处理步骤。 但是我得到一个错误,说writer元素既没有'writer'属性也没有元素。 我经历了链接spring batch:没有ItemWriter的Tasklet。 但是无法解决问题。 有人能告诉我我提到的代码段中要进行的具体更改吗? <batch:job id="helloWorldJob"> <batch:step id="step1"> <batch:tasklet> <batch:chunk reader="cvsFileItemReader" commit-interval="10"> </batch:chunk> </batch:tasklet> </batch:step> </batch:job> <bean id="cvsFileItemReader" class="org.springframework.batch.item.file.FlatFileItemReader"> <property name="resource" value="classpath:cvs/input/report.csv" /> <property name="lineMapper"> <bean class="org.springframework.batch.item.file.mapping
  • 我们可以在没有ItemReader和ItemWriter的情况下编写Spring Batch作业吗(Can we write a Spring Batch Job Without ItemReader and ItemWriter)
    问题 在我的项目中,我用Spring Batch 2.2编写了一个Quartz调度程序。 根据我的要求,我想运行一个调度程序以获取应用程序的config属性,以刷新所有GlassFish群集上的配置缓存。 因此,我不需要用于文件读/写操作的ItemWriter和ItemReader。 那么我可以从中删除ItemReader和ItemWriter吗? 我的工作的配置在下面提到: <batch:job id="reportJob"> <batch:step id="step1"> <batch:tasklet> <!--I want to remove ItemReader and ItemWriter as its not used --> <batch:chunk reader="ItemReader" writer="ItemWriter" commit-interval="10"> </batch:chunk> </batch:tasklet> </batch:step> <batch:listeners> <batch:listener ref="simpleListener"/> </batch:listeners> </batch:job> <bean id="jobDetail" class="org.springframework.scheduling.quartz
  • Spring Batch-将ItemWriter与列表一起使用(Spring Batch - Using an ItemWriter with List of Lists)
    问题 我们的处理器将List<?> (有效地传递List<List<?>> )返回到ItemWriter 。 现在,我们观察到JdbcBatchItemWriter没有被编程为处理item instanceof List 。 我们还观察到要处理List项目instance; 我们需要编写一个自定义的ItemSqlParameterSourceProvider 。 但不幸的是,它返回SqlParameterSource它只能处理一个item ,再没有能够处理List 。 因此,有人可以帮助我们了解如何处理JdbcBatchItemWriter中的列表列表吗? 回答1 通常,设计模式为: Reader -> reads something, returns ReadItem Processor -> ingests ReadItem, returns ProcessedItem Writer -> ingests List<ProcessedItem> 如果处理器返回List<Object> ,则需要Writer期望List<List<Object>> 。 您可以通过将JdbcBatchItemWriter作为委托包装在看起来像这样的ItemWriter中来实现: public class ListUnpackingItemWriter<T> implements ItemWriter
  • 在 Spring Batch Step、Tasklet 或 Chunks 之间做出决定(Deciding between Spring Batch Step, Tasklet or Chunks)
    问题 我有一个直接的要求,我需要读取一个项目列表(来自 DB)并需要处理这些项目,一旦处理,它必须更新到 DB。 我正在考虑将 Spring 批处理块与读取器、处理器和写入器一起使用。 我的读者将从列表中一次返回一个项目并将其发送到处理器,一旦处理结束,它就会返回到 Writer 并在那里更新数据库 稍后我可能会在这些方法中使用一些同步成本对其进行多线程处理。 在这里,我预见了一些担忧。 要处理的项目数量可能更多。 可能在 10,000 甚至更多。 处理器中需要进行一些逻辑计算。 因此一次处理 1 个项目。 即使它是具有 10 个线程的多线程,也不确定性能。 Writer 可以更新 DB 中该处理项目的结果。 不确定如何进行批量更新,因为它始终只有 1 个项目已处理和准备就绪。 这种方法对于这种用例是否正确,或者可以做任何更好的事情? 有没有其他方法可以一次调用读取器、处理器和写入器来处理一堆项目? 如果是这样,我是否需要创建一些机制,从列表中提取 10 个项目并将其提供给处理器? 似乎作者会在每条记录出现时更新,只有当作者收到一堆处理过的项目时,批量更新才有意义。 有什么建议吗? 请在此设计上点亮一些灯以获得更好的性能。 谢谢, 回答1 Spring Batch 是满足您需求的完美工具。 面向块的步骤允许您使用 commit-interval 属性配置要读取/处理/写入的项目数量
  • Create new output file using FlatFileItemWriter in spring-batch
    I have a simple spring batch job - read a file line by line, do something with the input string, and write some output. Output file contains every line of input plus some processing status for that line (success/failure.) The reads a file from: <dir>/<inputFolder>/<inputFileName> and writes processed output to <dir>/<outputFolder>/<inputFileName> All these values are passed as jobParameters File Reader is like so: <bean id="itemReader" class="org.springframework.batch.item.file.FlatFileItemReader" scope="step"> <property name="resource" value="file:#{jobParameters['cwd']}/#{jobParameters[
  • TransactionRequiredException: no transaction is in progress while using JPAItemWriter
    I am facing weird issue with an application which is a spring boot application. Details Here: The app has a spring batch job which uses JpaItemWriter for performing the writes to the database. Also, the application is configured to use hibernate ItemWriter configuration as follows: @Bean(name = "itemWriter") @StepScope public ItemWriter<Record> itemWriter() { JpaItemWriter<Record> itemWriter = new JpaItemWriter<>(); itemWriter.setEntityManagerFactory(emf); return itemWriter; } The batch job used to work great, but recently we upgraded our Jenkins to Jenkins(2.73.3). After that, the code built
  • Spring batch : FlatFileItemWriter header never called
    I have a weird issue with my FlatFileItemWriter callbacks. I have a custom ItemWriter implementing both FlatFileFooterCallback and FlatFileHeaderCallback. Consequently, I set header and footer callbacks in my FlatFileItemWriter like this : ItemWriter Bean @Bean @StepScope public ItemWriter<CityItem> writer(FlatFileItemWriter<CityProcessed> flatWriter, @Value("#{jobExecutionContext[inputFile]}") String inputFile) { CityItemWriter itemWriter = new CityItemWriter(); flatWriter.setHeaderCallback(itemWriter); flatWriter.setFooterCallback(itemWriter); itemWriter.setDelegate(flatWriter); itemWriter
  • Multi-threaded acces to Job Scope beans in Spring Batch 3.0
    In Spring Batch 3.0 I'm trying to use the new Job Scope functionality for beans in both partitioned and multi-threaded steps (configured with an task:executor bean), and in both cases I'm getting the exception Caused by: java.lang.IllegalStateException: No context holder available for job scope at org.springframework.batch.core.scope.JobScope.getContext(JobScope.java:153) at org.springframework.batch.core.scope.JobScope.get(JobScope.java:92) at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:338) but if i make the beans step scope it works OK. I
  • Spring Batch 3.0中对Job Scope Bean的多线程访问(Multi-threaded acces to Job Scope beans in Spring Batch 3.0)
    问题 在Spring Batch 3.0中,我试图在分区和多线程步骤(使用task:executor bean配置)中为bean使用新的Job Scope功能,在两种情况下,我都遇到了异常 Caused by: java.lang.IllegalStateException: No context holder available for job scope at org.springframework.batch.core.scope.JobScope.getContext(JobScope.java:153) at org.springframework.batch.core.scope.JobScope.get(JobScope.java:92) at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:338) 但是,如果我将bean步骤范围设置为正常,则可以。 我注意到JobSynchronizationManager上的评论说 注意,每个{@link Job}实现的责任是确保{@link JobContext}在可能与作业执行有关的每个线程(包括池中的工作线程)上可用。 所以我想知道是否需要做一些设置来设置它
  • Spring Batch:一个读取器,多个处理器和写入器(Spring Batch: One reader, multiple processors and writers)
    问题 在Spring批处理中,我需要将ItemReader读取的项目传递给两个不同的处理器和writer。 我想要实现的是... +---> ItemProcessor#1 ---> ItemWriter#1 | ItemReader ---> item ---+ | +---> ItemProcessor#2 ---> ItemWriter#2 之所以需要这样做,是因为与ItemWriter#2编写的项目相比,ItemWriter#1编写的项目应以完全不同的方式进行处理。 而且,ItemReader从数据库读取项目,并且它执行的查询在计算上非常昂贵,以至于两次执行相同的查询都应该被丢弃。 关于如何实现这种设置的任何提示? 或至少在逻辑上等效的设置? 回答1 如果您的商品应由处理器#1和处理器#2处理,则此解决方案有效 您必须使用以下签名创建处理器#0: class Processor0<Item, CompositeResultBean> 其中CompositeResultBean是定义为 class CompositeResultBean { Processor1ResultBean result1; Processor2ResultBean result2; } 在您的#0处理器中,将工作委托给#1和#2处理器,然后将结果放入CompositeResultBean
  • How can you restart a failed spring batch job and let it pick up where it left off?
    According to the Spring Batch documentation restarting of a job is supported out of the box but I cannot get it to start from where it left of. e.g. If my step processed 10 records it should start at record 11 with processing whenever I restart it. In practice this doesn't happen. It reads from the beginnen en reprocesses everything. Does anybody have a Java config based configuration of a simple job that reads a delimited file and writes the content to a db table that can be restarted from the point it stopped? @Configuration public class BatchConfiguration { @Value("${spring-batch
  • Spring Batch如何在将重复的项目发送到ItemWriter之前对其进行过滤(Spring Batch how to filter duplicated items before send it to ItemWriter)
    问题 我读取了一个平面文件(例如,每个用户每行1行的.csv文件,例如: UserId; Data1; Date2 )。 但是如何处理阅读器中的重复用户项(这里没有以前阅读过的用户列表...) stepBuilderFactory.get("createUserStep1") .<User, User>chunk(1000) .reader(flatFileItemReader) // FlatFileItemReader .writer(itemWriter) // For example JDBC Writer .build(); 回答1 过滤通常使用ItemProcessor完成。 如果ItemProcessor返回null,则该项目将被过滤并且不会传递给ItemWriter 。 否则,是的。 就您而言,您可以在ItemProcessor保留以前见过的用户列表。 如果以前未曾见过用户,请继续传递。 如果以前已经看到过,则返回null。 您可以在以下文档中阅读有关使用ItemProcessor进行过滤的更多信息:http://docs.spring.io/spring-batch/trunk/reference/html/readersAndWriters.html#filiteringRecords /** * This implementation assumes that
  • 春季批处理中的多个项目编写器(Multiple itemwriters in Spring batch)
    问题 我目前正在编写一个Spring批处理,在其中读取大量数据,进行处理,然后将这些数据传递给2个编写器。 一个编写器将简单地更新数据库,而第二个编写器将写入一个csv文件。 我打算编写自己的自定义编写器,并将两个itemWriter注入customItemWriter中,并在customItemWriter的write方法中调用两个项目编写器的write方法。 这种方法正确吗? 是否有任何可用的ItemWriter实现满足我的要求? 提前致谢 回答1 您可以使用Spring的CompositeItemWriter并将其委托给所有编写者。 这是一个配置示例。 回答2 您不必像示例一样使用xml。 如果您的其余代码使用注释,则只需执行以下操作。 public ItemWriter<T> writerOne(){ ItemWriter<T> writer = new ItemWriter<T>(); //your logic here return writer; } public ItemWriter<T> writerTwo(){ ItemWriter<T> writer = new ItemWriter<T>(); //your logic here return writer; } public CompositeItemWriter<T>
  • How to write a spring batch step without an itemwriter
    I am trying to configure a spring batch step without an item writer using below configuraion. However i get error saying that writer element has neither a 'writer' attribute nor a element. I went through the link spring batch : Tasklet without ItemWriter. But could not resolve issue. Could any one tell me the specific changes to be made in the code snippet I mentioned <batch:job id="helloWorldJob"> <batch:step id="step1"> <batch:tasklet> <batch:chunk reader="cvsFileItemReader" commit-interval="10"> </batch:chunk> </batch:tasklet> </batch:step> </batch:job> <bean id="cvsFileItemReader" class=
  • Spring Batch Multiple Threads
    I am writing a Spring Batch with idea of scaling it when required. My ApplicationContext looks like this @Configuration @EnableBatchProcessing @EnableTransactionManagement @ComponentScan(basePackages = "in.springbatch") @PropertySource(value = {"classpath:springbatch.properties"}) public class ApplicationConfig { @Autowired Environment environment; @Autowired private JobBuilderFactory jobs; @Autowired private StepBuilderFactory stepBuilderFactory; @Bean public Job job() throws Exception { return jobs.get("spring_batch") .flow(step()).end() .build(); } @Bean(name = "dataSource", destroyMethod =
  • load external config file in spring batch admin
    I downloaded the spring batch admin application in the github repo and I imported it in eclipse. it works perfectly. Then, I asked myself how to import an external config file into the application that I can use in my job definition class. I tried this : VM Arguments -Dspring.config.location=C:/path/to/config/file/application.properties Job configuration src/main/java org.springframework.batch.admin.sample.job ------------------------------------------- @Configuration public class JobConfiguration { //I try to import this properties from an external config file. @Value("${folder.input.files}")
  • 在Spring Batch中,我们如何在作业的不同步骤之间共享数据?(How can we share data between the different steps of a Job in Spring Batch?)
    问题 深入研究Spring Batch,我想知道我们如何在Job的不同步骤之间共享数据? 我们可以为此使用JobRepository吗? 如果是,我们该怎么做? 还有其他方法可以做到/实现吗? 回答1 作业存储库可间接用于在步骤之间传递数据(Jean-Philippe正确的做法是,最好的方法是将数据放入StepExecutionContext ,然后使用StepExecutionContext ExecutionContextPromotionListener将步骤执行上下文键提升为JobExecutionContext 。 需要注意的是,还有一个侦听器也可以将JobParameter键提升为StepExecutionContext (更详细地命名为JobParameterExecutionContextCopyListener ); 如果您的工作步骤彼此不完全独立,则会发现您经常使用这些工具。 否则,您将不得不使用更加精细的方案在步骤之间传递数据,例如JMS队列或(禁止使用天堂)硬编码的文件位置。 至于在上下文中传递的数据大小,我还建议您将其保持较小(但我没有关于 回答2 从一个步骤中,您可以将数据放入StepExecutionContext 。 然后,向监听器,您可以促进数据从StepExecutionContext到JobExecutionContext 。