Linux on paper.archive

Login with an ssh key

Thu, 11 Aug 2016 16:07:00 +0000

Set up an ssh key on macOS so you can log into the server without typing a password
~~Normally I don’t do this, for security.~~

macOS

- Generate a key

`1`	`ssh-keygen -t rsa -C "name"`

- View the public key

`1`	`cat ~/.ssh/id_rsa.pub`

Target Server (CentOS)

- Add the public key to the `~/.ssh/authorized_keys` file

`1`	`vi ~/.ssh/authorized_keys`

Logging into an AWS EC2 server with a password

Fri, 08 Apr 2016 11:33:00 +0000

AWS ec2 servers block ssh access with a password by default. Let’s edit the file below to allow it.

Done on centos 6.5

1
2

# 루트계정에 패스워드 지정
passwd root

sudo vi /etc/ssh/sshd_config

# 아래 두개 옵션을 yes로 하고 저장후 닫기 
PermitRootLogin yes
PasswordAuthentication yes

# sshd 재시작 
service sshd restart

Switching Flume's default logging from log4j to logback

Fri, 30 Oct 2015 18:13:00 +0000

Writing Flume log files with logback

How to install logback in Flume

Download logback from (http://logback.qos.ch/download.html). (currently v1.1.3)
Extract it and copy logback-classic-1.1.3.jar and logback-core-1.1.3.jar into $FLUME_HOME/lib.
For the existing log4j, rename ./lib/slf4j-log4j12-1.6.1.jar to ./lib/slf4j-log4j12-1.6.1.jar.back. Removing log4j is optional (if you leave it, both will write logs)
Edit logback.xml and put it at $FLUME_HOME/conf/logback.xml.

logback.xml sample

<?xml version="1.0" encoding="UTF-8"?>
<configuration>

	<!-- Appenders -->
	<appender name="console" class="ch.qos.logback.core.ConsoleAppender">    
      	<encoder>
        	<pattern>[%-5level] %d{HH:mm:ss.SSS} [%thread] %logger{36} - %msg%n</pattern>
       	</encoder>
    </appender>
    <appender name="daily" class="ch.qos.logback.core.rolling.RollingFileAppender">
        <prudent>false</prudent>
        <file>./logs/flume1.log</file>
        <rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
            <fileNamePattern>./logs/old/flume1.%d{yyyy-MM-dd}.%i.log</fileNamePattern>
            <timeBasedFileNamingAndTriggeringPolicy class="ch.qos.logback.core.rolling.SizeAndTimeBasedFNATP">
                <maxFileSize>100mb</maxFileSize>
            </timeBasedFileNamingAndTriggeringPolicy>
            <maxHistory>30</maxHistory>
        </rollingPolicy>
        <encoder>
            <pattern>[%-5level] %d{HH:mm:ss.SSS} %logger{36} - %msg%n</pattern>
        </encoder>
    </appender>

    <appender name="event" class="ch.qos.logback.core.rolling.RollingFileAppender">
        <prudent>false</prudent>
        <file>./logs/collect.log</file>
        <rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
            <fileNamePattern>./logs/old/collect.%d{yyyy-MM-dd}.%i.log</fileNamePattern>
            <timeBasedFileNamingAndTriggeringPolicy class="ch.qos.logback.core.rolling.SizeAndTimeBasedFNATP">
                <maxFileSize>100mb</maxFileSize>
            </timeBasedFileNamingAndTriggeringPolicy>
            <maxHistory>30</maxHistory>
        </rollingPolicy>
        <encoder>
            <pattern>%msg%n</pattern>
        </encoder>
    </appender>

	<logger name="kimpaper" level="debug">
        <appender-ref ref="event" />
	</logger>

    <!-- 3rdparty Loggers -->
	<logger name="org.apache.flume.lifecycle" level="info">
	</logger>
	<logger name="org.jboss" level="warn">
	</logger>
	<logger name="org.mortbay" level="info">
	</logger>
	<logger name="org.apache.avro.ipc.NettyTransceiver" level="warn">
	</logger>
	
	<logger name="org.apache.hadoop" level="info">
	</logger>
	<logger name="org.apache.hadoop.hive" level="error">
	</logger>
	
	
	<!-- Root Logger -->
	<root level="info">
		<appender-ref ref="console" />
		<appender-ref ref="daily" />
	</root>
	
</configuration>

Configure logback to record events

I was going to use file_roll as the sink that collects and stores logs on the master server, but it has downsides.

Flume install and test (mac)

Thu, 29 Oct 2015 15:51:00 +0000

What is Flume?

A service that collects log files written across multiple machines (multiple services..) into one place in real time

Install

Download from http://flume.apache.org/download.html
Extract it somewhere suitable ~/dev/tool/flume
If JAVA_HOME is not set, open ~/.bash_profile and set the environment variable.

`1`	`export JAVA_HOME = /usr (자바 경로.)`

I installed and tested on mac, but I trust it works fine on centos too.

Basic flume.conf file

# The configuration file needs to define the sources,
# the channels and the sinks.
# Sources, channels and sinks are defined per agent,
# in this case called 'agent'

agent.sources = seqGenSrc
agent.channels = memoryChannel
agent.sinks = loggerSink

# For each one of the sources, the type is defined
agent.sources.seqGenSrc.type = seq

# The channel can be defined as follows.
agent.sources.seqGenSrc.channels = memoryChannel

# Each sink's type must be defined
agent.sinks.loggerSink.type = logger

#Specify the channel the sink should use
agent.sinks.loggerSink.channel = memoryChannel

# Each channel's type is defined.
agent.channels.memoryChannel.type = memory

# Other config values specific to each type of channel(sink or source)
# can be defined as well
# In this case, it specifies the capacity of the memory channel
agent.channels.memoryChannel.capacity = 100

Reference) https://flume.apache.org/FlumeUserGuide.html The config file is divided into sources, channels, and sinks.

Setting up a server on DigitalOcean (CentOS 6.7)

Tue, 15 Sep 2015 20:40:00 +0000

Set up a server for a simple messaging app I’m building with a friend. Of course I don’t really know what I’m doing. Everything comes from Google..

1. Allocate SWAP memory

https://www.digitalocean.com/community/tutorials/how-to-add-swap-on-centos-6

dd if=/dev/zero of=/swapfile bs=1024 count=2048k
mkswap /swapfile
swapon /swapfile
chown root:root /swapfile
chmod 0600 /swapfile

Append the following to /etc/fstab.

`1`	`/swapfile swap swap defaults 0 0`

2. Change server time to KST & install rdate

1
2

ln -sf /usr/share/zoneinfo/Asia/Seoul /etc/localtime
yum install rdate

3. Install Java 1.7 via rpm

http://www.oracle.com/technetwork/java/javase/downloads/jre7-downloads-1880261.html Download the 64-bit Linux version from the link above.

Korean fonts in JFreeChart (CentOS font setup)

Wed, 01 Jul 2015 20:33:00 +0000

I’m using JFreeChart, and when it runs on Tomcat, Korean text renders as broken boxes.

Put the font files in /usr/share/fonts and run fc-cache -fv.

// 코드 상에 아래와 같이 폰트를 지정한다.
private static final Font _BASE_FONT = new Font("나눔고딕",Font.PLAIN,11);
...
중략
'''
// setFont를 적절하게 코드 내에 사용한다.
lineAndShapeRenderer.setLegendTextFont(i, _BASE_FONT);
...

No Tomcat restart needed.

When Apache returns 403 errors

Fri, 27 Mar 2015 01:47:00 +0000

If you’ve tried everything and still keep getting 403..

`1`	`chcon -R --reference=/var/www /www/webroot`

`1`	`chcon -R -h -t httpd_sys_content_t /www/webroot`

git command collection

Wed, 04 Mar 2015 17:33:00 +0000

svn to git migration

1
2

git svn clone --stdlayout --no-metadata -A users.txt svn://example.com/repository/projectname
cd projectname

The command below is supposed to generate users.txt, but it doesn’t work for me.. maybe because perl is missing..
svn log ^/ --xml | grep -P "^<author" | sort -u | \ perl -pe 's/<author>(.*?)<\/author>/$1 = /' > users.txt

handle ignore file

`1`	`git svn show-ignore -i trunk > .gitignore`

set remote git

`1`	`git remote add origin git@git.example.com:group/projectname.git`

handle tags

`1`	`git for-each-ref refs/remotes/tags \| cut -d / -f 4- \| grep -v @ \| while read tagname; do git tag "$tagname" "tags/$tagname"; git branch -r -d "tags/$tagname"; done`

handle branches

`1`	`git for-each-ref refs/remotes \| cut -d / -f 3- \| grep -v @ \| while read branchname; do git branch "$branchname" "refs/remotes/$branchname"; git branch -r -d "$branchname"; done`

push

1
2

git push origin --all
git push origin --tags

revert local commit

`1`	`git reset —hard remotes/origin/HEAD`

backup

`1`	`sudo gitlab-rake gitlab:backup:create`

restore (restores the most recent backup)

`1`	`sudo gitlab-rake gitlab:backup:restore`

restore OPTION (I guess it restores to the timestamp specified below.. haven’t tried it..)
BACKUP=timestamp_of_backup (required if more than one backup exists)

Unicode (utf8mb4) support in MySQL and MariaDB

Tue, 10 Feb 2015 20:48:00 +0000

1. Check the current settings

MariaDB [(none)]> show variables like "%character%";show variables like "%collation%";
 +--------------------------+---------------------------------+
 | Variable_name            | Value                           |
 +--------------------------+---------------------------------+
 | character_set_client     | utf8                            |
 | character_set_connection | utf8                            |
 | character_set_database   | utf8                            |
 | character_set_filesystem | binary                          |
 | character_set_results    | utf8                            |
 | character_set_server     | utf8                            |
 | character_set_system     | utf8                            |
 | character_sets_dir       | /usr/local/mysql/share/charsets/|
 +--------------------------+---------------------------------+

2. sudo vi /etc/my.cnf and edit the following parts

[client]
default-character-set=utf8mb4

[mysql]
default-character-set=utf8mb4

[mysqld]
collation-server = utf8mb4_unicode_ci
character-set-server = utf8mb4

3. Restart the service

`1`	`service mysql restart`

4. Verify

MariaDB [(none)]> show variables like "%character%";show variables like "%collation%";

+--------------------------+----------------------------+
| Variable_name            | Value                      |
+--------------------------+----------------------------+
| character_set_client     | utf8mb4                    |
| character_set_connection | utf8mb4                    |
| character_set_database   | utf8mb4                    |
| character_set_filesystem | binary                     |
| character_set_results    | utf8mb4                    |
| character_set_server     | utf8mb4                    |
| character_set_system     | utf8                       |
| character_sets_dir       | /usr/share/mysql/charsets/ |
+--------------------------+----------------------------+
8 rows in set (0.00 sec)

+----------------------+--------------------+
| Variable_name        | Value              |
+----------------------+--------------------+
| collation_connection | utf8mb4_general_ci |
| collation_database   | utf8mb4_unicode_ci |
| collation_server     | utf8mb4_unicode_ci |
+----------------------+--------------------+
3 rows in set (0.00 sec)

Finally, you need to change the type of the actual columns.

CentOS command collection

Thu, 05 Feb 2015 17:42:00 +0000

#프로그램 설치위치 찾기
which java
/usr/bin/java

#오래된 로그파일만 찾아서 삭제함
find /logs/httpd/ -type f -mtime +180 -exec rm -f {} \;

#hosts 편집
vi /etc/hosts

#dns서버 수정
vi /etc/resolv.conf

#OS 비트 확인
getconf LONG_BIT

#OS 버전 확인
cat /etc/issue

#폰트목록
fc-list

#폰트 반영 (/usr/share/fonts 에 폰트 넣고)
fc-cache -fv

#로그보기
tail -f file.log -n 1000

#crond 명령
# 목록보기
crontab -l
# 편집
crontab -e 

# 톰켓 프로세스 확인
ps -ef | grep tomcat

# 프로세스 kill
kill -9 {PID}

When Tomcat keeps throwing OutOfMemory

Thu, 05 Feb 2015 17:10:00 +0000

Add the following at the very top of catalina.sh to increase the memory. (Careful — keep the server’s total memory in mind.)

export CATALINA_OPTS="-Djava.awt.headless=true -server -Xms2048m -Xmx2048m -XX:NewSize=256m -XX:MaxNewSize=256m -XX:PermSize=256m -XX:MaxPermSize=512m -XX:+CMSClassUnloadingEnabled -XX:+CMSPermGenSweepingEnabled"

Reference) http://stackoverflow.com/questions/88235/dealing-with-java-lang-outofmemoryerror-permgen-space-error