MapReduce Framework Optimization

Modeling and Optimization of Straggling Mappers

ABSTRACT
MapReduce framework is widely used to parallelize batch jobs since it exploits a high degree of multi-tasking to process them. However, it has been observed that when the number of mappers increases, the map phase can take much longer than expected. This paper analytically shows that stochastic behavior of mapper nodes has a negative effect on the completion time of a MapReduce job, and continuously increasing the number of mappers without accurate scheduling can degrade the overall performance. We analytically capture the effects of stragglers (delayed mappers) on the performance. Based on an observed delayed exponential distribution (DED) of the response time of mappers, we then model the map phase by means of hardware, system, and application parameters. Mean sojourn time (MST), the time needed to sync the completed map tasks at one reducer, is mathematically formulated. Following that, we optimize MST by finding the task inter-arrival time to each mapper node. The optimal mapping problem leads to an equilibrium property investigated for different types of inter-arrival and service time distributions in a heterogeneous datacenter (i.e., a datacenter with different types of nodes). Our experimental results show the performance and important parameters of the different types of schedulers targeting MapReduce applications. We also show that, in the case of mixed deterministic and stochastic schedulers, there is an optimal scheduler that can always achieve the lowest MST.

Comments

Post a Comment

Popular posts from this blog

System and methods for UICC-based secure communication

ABSTRACT A system that incorporates the subject disclosure may include, for example, instructions which when executed cause a device processor to perform operations comprising sending a service request to a remote management server; receiving from the management server an authentication management function and an encryption key generator for execution by a secure element and an encryption engine for execution by a secure device processor, sending a request to establish a communication session with a remote device; and communicating with the remote device via a channel established using an application server. The secure element and the secure device processor authenticate each other using a mutual authentication keyset. The secure element, the secure device processor and the device processor each have a security level associated therewith; the security level associated with the secure device processor is intermediate between that of the secure element and that of the device processor....
Read more

Computer Vision Keywords

2D attacks 2D Face 2D Face Authentication 3D blood flow estimation 3D Face 3D Face Localisation 3D face model 3D face recognition 3D localization 3D Masks 3D model base indexing 3D object recognition abnormality detection accelerated exact k-means Accent Identification Accented speech Accentual mismatch accessible games acoustic frequency Acoustic model adaptation acoustic modeling Acoustic reverberation sparsity models Acoustic reverberation structures Acoustic signal processing acoustic stream acquisition ACT metric activation function active shape model activity Actor–critic Ad hoc array calibration Ad hoc microphone array calibration Adaboost adalaine adaptive exploration/exploitation trade-off adaptive learning rate adaptive resonance theory adaptive steepness Addressee estimation Adequacy of diffuseness Ad-hoc microphone array Adhoc microphone array calibration Ad-hoc microphone array calibration Ad-hoc microphone calibration AFC A...
Read more